<html>
<head><meta charset="utf-8"><title>Nt tokens in the parser · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html">Nt tokens in the parser</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="174630189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174630189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174630189">(Aug 31 2019 at 19:34)</a>:</h4>
<p>I would like to move rust’s parser to the same token model that proc macros use. However, the current model contains various Nt tokens which store parsed fragments of AST. My understanding is that they exist solely for macro-by-example macros.</p>
<p>Are such token fundamental, or are we hoping to replace them with token trees with none delimiters in the future?</p>



<a name="174639568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639568">(Sep 01 2019 at 00:43)</a>:</h4>
<p>Not fundamental, but we may need some hacks for compatibility.</p>



<a name="174639628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639628">(Sep 01 2019 at 00:44)</a>:</h4>
<p><code>$expr:expr</code> is a <code>TokenTree::Group</code> with an implicit delimiter in the token model.</p>



<a name="174639639"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639639" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639639">(Sep 01 2019 at 00:45)</a>:</h4>
<p>I think this model covers most of the existing behavior, but not all.</p>



<a name="174639651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639651">(Sep 01 2019 at 00:45)</a>:</h4>
<p>See <a href="https://github.com/rust-lang/rust/issues/55414" target="_blank" title="https://github.com/rust-lang/rust/issues/55414">https://github.com/rust-lang/rust/issues/55414</a>, <a href="https://github.com/rust-lang/rust/issues/47358" target="_blank" title="https://github.com/rust-lang/rust/issues/47358">https://github.com/rust-lang/rust/issues/47358</a> for exceptions.</p>



<a name="174639705"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639705" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639705">(Sep 01 2019 at 00:47)</a>:</h4>
<p>The basic example is</p>
<div class="codehilite"><pre><span></span><span class="n">macro_rules</span><span class="o">!</span><span class="w"> </span><span class="n">doc_from_file</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="p">(</span><span class="cp">$expr</span>:<span class="nc">expr</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="cp">#[doc = $expr]</span><span class="w"></span>
<span class="w">        </span><span class="k">struct</span> <span class="nc">S</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="p">};</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="n">doc_from_file</span><span class="o">!</span><span class="p">(</span><span class="n">include_str</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;doc.md&quot;</span><span class="p">));</span><span class="w"></span>
</pre></div>



<a name="174639750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639750">(Sep 01 2019 at 00:48)</a>:</h4>
<p><code>$expr</code> is a nonterminal token with an <code>ast::Expr</code> fragment inside it.</p>



<a name="174639765"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639765" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639765">(Sep 01 2019 at 00:49)</a>:</h4>
<p>And that <code>ast::Expr</code> is visited by the visitor expanding macros as any other expression in less exotic position.</p>



<a name="174639775"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639775" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639775">(Sep 01 2019 at 00:49)</a>:</h4>
<p>So we get <code>#[doc = "contents of doc.md"]</code> after macro expansion.</p>



<a name="174639832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639832">(Sep 01 2019 at 00:50)</a>:</h4>
<p>This is entirely incompatible with the token model where <code>include_str!("doc.md")</code> is not an <code>ast::Expr</code>, but just some token stream that fits into expression grammar.</p>



<a name="174639904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639904">(Sep 01 2019 at 00:52)</a>:</h4>
<hr>
<p>(I'm currently preparing an experiment for checking whether this behavior can be limited to e.g. only values in key-value attributes, rather than to arbitrary non-terminal tokens.)</p>



<a name="174639913"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/Nt%20tokens%20in%20the%20parser/near/174639913" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Vadim Petrochenkov <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/Nt.20tokens.20in.20the.20parser.html#174639913">(Sep 01 2019 at 00:53)</a>:</h4>
<hr>
<p>There may be other cases where nonterminals are fundamentally treated as AST pieces and not tokens streams, but I'm not aware of them right now.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>